AI Test Toolkit: language-first data-driven eval support#9460
Draft
thloke wants to merge 10 commits into
Draft
Conversation
…ce) support Adds a shared ITestDataSource provider, an 'AIT Test Case Context' interface (extends ITestContext) and a delegating context, a per-case dataset-lineage bridge for logging, dual-mode dataset resolution, and coexistence with the legacy per-row expansion (auto-detected via CodeUnit Metadata.HasTestDataSource, with a Language First line flag as override). Docs updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n pattern Adds a step-by-step migration recipe (attribute + context-parameter + drop the AIT Test Context var), before/after example, rules (whole-codeunit, auto-detect), variations (multi-turn, harms/adversarial, custom providers) and a checklist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrating AIT evals to the language-first [TestDataSource] construct moves the per-case fan-out to the platform test runner, so the toolkit's classic per-case bracketing (Test Runner - Mgt.OnBefore/OnAfterTestMethodRun subscribers, which only run under an AIT suite) no longer engages. This adds the platform-native seam: - codeunit 149050 "AIT Test Handler" implements System.Testability.ITestHandler. OnBeforeTestCaseRun resets per-case accuracy/turns/token accounting and opens the run-procedure output scope; OnAfterTestCaseRun records the case outcome. Consuming eval codeunits opt in via TestHandlers = "AIT Test Handler". - enumextension 149035 registers it under the platform TestHandler enum. - "AIT DD Current Case" now buffers the per-case identity/outcome (in memory, so it survives the function isolation rollback) and flushes one AIT Log Entry from the out-of-isolation OnAfterTestMethodRun seam (decision C8-A). Note: that seam is raised by the Test Runner - Mgt runner (AL Test Tool / Test Suite Mgt); the pure platform test-execution path (al CLI) does not raise it, so log persistence there is a tracked follow-up. - "AIT Test Suite Mgt.".AddDataDrivenLogEntry writes a per-case log entry with a session-scoped Run ID when there is no AIT suite; "AIT Test Run Iteration". IsRunningUnderAITSuite gates the handler off under an AIT suite to avoid double logging. - Register the new objects in the toolkit object permission set and document the TestHandlers step in the migration guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AddDataDrivenLogEntry now calls AgentTestContextImpl.LogAgentTasks after inserting the AIT Log Entry, matching the classic AddLogEntry path so agent-eval task correlation is preserved for language-first evals running on the platform runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion) Now that the platform runs ITestHandler hooks outside the per-function isolation scope, the handler can write the AIT Log Entry directly in OnAfterTestCaseRun and it persists past the case rollback. Remove the buffer/flush workaround: the pending-case state and the OnAfterTestMethodRun flush subscriber in AIT DD Current Case, and AIT DD Current Case is reduced to the dataset-lineage bridge + RunId + case-start baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Could not find a linked ADO work item. Please link one by using the pattern 'AB#' followed by the relevant work item number. You may use the 'Fixes' keyword to automatically resolve the work item when the pull request is merged. E.g. 'Fixes AB#1234' |
…osed dataset Uptakes the reshaped platform ITestDataSource contract (enumerate + retrieve): - AIT Test Data Source: GetDataRows(List of [ITestContext]) -> ListTestCases(): List of [Text] (enumerate all dataset row codes up front) + GetTestCase(index, id): interface ITestContext (materialize one row's context on demand, only for cases that actually run). - AIT Test Case Context: declare Identifier() explicitly now that the base platform ITestContext is a pure marker, preserving the eval-facing API. - Fail-closed (D6): an unresolvable dataset id, or a dataset that resolves to zero rows, now errors instead of silently producing zero cases (which the runtime would treat as a false-green pass). Compiles cleanly (76 files) against the reshaped System symbols + updated compiler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…only (D8) Language-first detection is now solely CodeUnit Metadata.\"Has Test Data Source\" (platform field 12). Removes the interim \"Language First\" line flag (AIT Test Method Line field 8, its page control, and the import/export XMLPort field-attribute) and the OR'd fallback in ExpandCodeunit. No backward compatibility to preserve (unreleased). Toolkit compiles cleanly against the reshaped symbols. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses three single-instance state bugs that dropped or mis-attributed per-case logs when a language-first eval runs on the platform test runner: - B2 (output lost): AddDataDrivenLogEntry read the per-case output from the handler's own (non-SingleInstance) "AIT Test Suite Mgt." instance, which is always empty. It now reads through the SingleInstance "AIT Test Context Impl." (new ConsumeRunProcedureOutput) - the same instance the test body wrote the output to. - B1 (stale lineage): the current case was bound lazily (only when the body first read its input), so a case that never touched its context logged the previous case's dataset lineage. GetTestCase now binds the current case eagerly at materialization (runs once per executing case, before the body and the OnAfterTestCaseRun log write). - B3 (leaked suite flag): AIT Test Run Iteration set ActiveAITTestSuite but never cleared it, so a standalone platform-runner data-driven run in the same session was mistaken for a suite run and the handler skipped its per-case logging. It is now cleared after each iteration. Toolkit compiles cleanly against the reshaped symbols. NOTE: deferred (need the full toolkit app stack to validate, or a design decision): the OnBeforeTestCaseRun scenario-start turn-count ordering, and the language-first credit-limit skip re-hook (the limit provider is an AIT-suite concept; standalone platform-runner runs have no suite). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ta-driven evals Re-hooks the credit-limit skip for language-first evals run on the platform test runner (no AIT suite). AIT Test Handler.OnBeforeTestCaseRun now checks the global monthly Copilot-credit limit (AIT Eval Monthly Copilot Cred., an environment/company budget - not suite-scoped) and, when enforcement is enabled and the limit is reached, calls Context.Skip() (reported Skipped by the platform runner) and writes a Skipped AIT Log Entry via the new LogSkippedDataDrivenEval (mirrors AddDataDrivenLogEntry with Status::Skipped + reason, keyed by RunId/codeunit/procedure/case; the existing LogSkippedEval is suite-only). Enforcement disabled (default) -> IsLimitReached is false -> no-op. Mirrors the classic app-suite behavior in AIT Test Run Iteration.OnBeforeTestMethodRun. Toolkit compiles cleanly against the reshaped symbols. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract shared helpers InitDataDrivenLogEntryHeader and EnrichLogEntryFromTestInput so AddLogEntry, AddDataDrivenLogEntry and LogSkippedDataDrivenEval no longer duplicate the header population and Test Input enrichment. Centralize the AutoIncrement "Entry No." := 0 idiom with an explanatory comment. Also fix the inverted duration guard in AddLogEntry (= 0DT -> <> 0DT) that the duplication had let drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7391b0-95a6-49b9-9a9a-14397106dd79
thloke
force-pushed
the
private/thloke/ait-language-first-datadriven
branch
from
July 17, 2026 02:53
2fb8a66 to
2b12d6c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds language-first (
[TestDataSource]) data-driven support to the AI Test Toolkit so AIT evals can run on the platform test runner, coexisting with the classic app-based Eval Suite runner. Companion to platform (BC-Platform) and compiler (BC-DeveloperExperience) PRs.Changes (all under
src/Tools/AI Test Toolkit)src/DataDriven/):AIT Test Case Contextinterface (extendsITestContext, exposes the classicAIT Test Contextsurface);AIT DD Test Context(implements it);AIT Test Data Source(implements ITestDataSource, resolves the dataset from the AIT run context or the attribute literal, one context perTest Inputrow);AIT Test Handler(implements ITestHandler) + enum registration;AIT DD Current Case(dataset-lineage bridge + session run id + per-case baseline).AIT Log Entryper case directly inOnAfterTestCaseRun(relies on the platform change that runs handlers outside function isolation).AddDataDrivenLogEntry+IsRunningUnderAITSuitegate so the classic suite path and the platform-runner path never double-log.CodeUnit Metadata."Has Test Data Source") and added once (no per-row expansion);Language Firstflag on the eval line as an explicit override.Migration surface for consumers
A typical eval migrates with three declarations and no new objects:
[TestDataSource(Codeunit::"AIT Test Data Source", '<dataset>')], acontext: interface "AIT Test Case Context"parameter, andTestHandlers = "AIT Test Handler".Validation
Validated via
al runtestsagainst a local BC 29.0 server (provider + context fan-out, handler logging). Depends on the platform/compiler PRs to land for full behavior; theContext.Skip()credit-limit re-hook and streaming Harms provider are follow-ups once those ship.Draft.